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© Adaptive interpolation is performed by apparatus 
operating upon a digitized image signal obtained 
from an image sensor having color photosites that 
generate a plurality of color values, but only one 
color per photosite. A digital processor obtains gra- 
dient values from the differences between luminance 
values in vertical and horizontal image directions. 
The gradient values are compared to a program- 
mable threshold in order to select one of the direc- 
tions as the preferred orientation for the interpolation 
of additional luminance values. The interpolation is 
then performed upon values selected to agree with 
the preferred orientation. 
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Field of Invention 

This invention pertains to the field of electronic 
imaging, in particular to the generation of color 
images. 5 

BACKGROUND OF THE INVENTION 

A single-sensor camera detects the spatially 
varying intensity of light at image locations cor- io 
responding to a regular pattern of pixel locations on 
the sensor. In a color single-sensor camera, a color 
filter array overlies the sensor such that the sensor 
detects the intensity of colors at varying pixel loca- 
tions according to a regular color filter array (CFA) 75 
pattern of, generally, three colors. Ordinarily, the 
CFA is a regular pattern of color filters for detecting 
only one color at each pixel location. Consequently, 
a single sensor color camera does not capture 
original data corresponding to all three colors for 20 
each pixel. Instead, it captures one color for each 
pixel, so that interpolation is required to construct 
three full color image planes for each image. 

A typical camera system generates red, green, 
and blue colors. A color filter array interpolation 25 
algorithm is used to convert the image from a 
sparsely sampled color image (one color per pixel) 
to a full red, green, blue (RGB) image (i.e., RGB for 
each pixel). Most color filter array patterns have a 
high-frequency signal that is sampled more regu- 30 
larly, and more frequently, in the pattern than the 
other colors. In an RGB image, this high frequency 
signal is green; it is also referred to as the lu- 
minance signal, which represents the higher fre- 
quency detail and the maximum sensitivity of the 35 
human eye. Ordinarily, traditional bilinear interpola- 
tion is used to generate a full green image plane. 
For instance, the green data from green pixels on 
either side of a red or blue pixel (i.e., a "missing 
green" pixel) are used to interpolate the "missing 40 
green" value for the red or blue location. Then, 
traditional bilinear interpolation of color difference 
signals, also called chrominance signals, is utilized 
to interpolate the other colors of the CFA pattern 
for each pixel. A traditional method of this type is 45 
disclosed in U.S. Patent No. 4,642,678. 

A problem with such traditional methods is that 
they are prone to colored edge artifacts in the 
image. This problem can be treated through use of 
more sophisticated interpolation techniques, such 50 
as described in U.S. Patent No. 4,630,307, which 
uses prior knowledge about features existing in the 
neighborhood. The image data is utilized to deter- 
mine the appropriate algorithm, and the missing 
data is reconstructed using the selected algorithm. 55 
For example, in the '307 patent, different interpola- 
tion routines are used for edges, stripes, and cor- 
ners. The particular feature is determined by com- 



paring the pixel data with templates stored in a 
computer. 

As pointed out before, the major shortcoming 
of the traditional procedures concerns the genera- 
tion of artifacts at color edges. The sophisticated 
procedure, exemplified by the *307 patent, can 
reduce these artifacts, but at considerable cost and 
complexity in processing capability. 

SUMMARY OF THE INVENTION 

An object of the present invention is to modify 
prior, sophisticated algorithms so as to reduce col- 
or edge artifacts and to improve image sharpness, 
without unduly increasing cost and complexity. 

A further object of the invention is to perform 
interpolation in a simplified, adaptive manner in 
order to reduce color edge artifacts, reduce noise 
in the image, and to improve image sharpness. 

The above-stated objects are realized by ap- 
paratus for processing a digitized image signal 
obtained from an image sensor having color 
photosites that generate a plurality of color values, 
but only one color value for each photosite loca- 
tion. Such apparatus includes 

a) means for storing the digitized image signal 
as an image-wise pattern of luminance and 
chrominance values, wherein some photosite lo- 
cations lack luminance values; and 

b) a processor operative with said storing means 
for generating a luminance value missing from a 
photosite location by the interpolation of an ad- 
ditional luminance value for such locations from 
luminance values at nearby photosite locations. 
In accordance with the invention the processor 
includes 

a) means for generating gradient values from 
the differences between luminance values in 
at least two image directions; 

b) means for comparing the gradient values 
to one or more thresholds; 

c) means responsive to the threshold com- 
parison for selecting one of said at least two 
image directions as a preferred orientation for 
the interpolation of an additional luminance 
value; and 

d) means for interpolating the additional lu- 
minance value from luminance values select- 
ed to agree with the preferred orientation. 

The principal advantage of the invention is that 
it can reduce color artifacts without adding undue 
complexity to the processing. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will be described in relation to 
the drawings, wherein: 
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Figure 1 is a block diagram of an electronic still 
camera employing interpolation processing ac- 
cording to the invention; 

Figure 2 is a block diagram of the interpolation 
processing technique used in connection with 
the invention; 

Figure 3 is a diagram of the Bayer geometry for 
a color filter array; and 

Figures 4A-4D show four phases of the Bayer 
geometry useful in explaining the interpolation 
technique. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

Since single-sensor electronic cameras em- 
ploying color filter arrays are well known, the 
present description will be directed in particular to 
elements forming part of, or cooperating more di- 
rectly with, apparatus and method in accordance 
with the present invention. Elements not specifi- 
cally shown or described herein may be selected 
from those known in the art. 

Referring initially to Figures 1 and 2, an elec- 
tronic still camera is divided generally into an input 
section 2 and an interpolation and recording sec- 
tion 4. The input section 2 includes an exposures 
section 10 for directing image light from a subject 
(not shown) toward an image sensor 12. Although 
not shown, the exposure section 10 includes con- 
ventional optics for directing the image light 
through a diaphragm, which regulates the optical 
aperture, and a shutter, which regulates exposure 
time. The sensor 12, which includes a two-dimen- 
sional array of photosites corresponding to picture 
elements of the image, is a conventional charge- 
coupled device (CCD) using either well-known in- 
terline transfer or frame transfer techniques. The 
sensor 12 is covered by a color filter array (CFA) 
13, known as the Bayer array, which is described 
in U.S. Patent No. 3,971,065 and herewith incor- 
porated by reference. The Bayer geometry is 
shown in Figure 3, wherein each color covers a 
photosite, or picture element (pixel), of the sensor. 
In particular, chrominance colors (red and blue) are 
interspersed among a checkerboard pattern of lu- 
minance colors (green). The sensor 12 is exposed 
to image light so that analog image charge in- 
formation is generated in respective photosites. 
The charge information is applied to an output 
diode 14, which converts the charge information to 
analog image signals corresponding to respective 
picture elements. The analog image signals are 
applied to an A/D converter 16, which generates a 
digital image signal from the analog input signal for 
each picture element. The digital signals are ap- 
plied to an image buffer 18, which may be a 
random access memory (RAM) with storage capac- 



ity for a plurality of still images. 

A control processor 20 generally controls the 
input section 2 of the camera by initiating and 
controlling exposure (by operation by the dia- 

5 phragm and shutter (not shown) in the exposure 
section 10), by generating the horizontal and verti- 
cal clocks needed for driving the sensor 12 and for 
clocking image information therefrom, and by en- 
abling the A/D converter 16 in conjunction with the 

w image buffer 18 for each signal segment relating to 
a picture element. (The control processor 20 would 
ordinarily include a microprocessor coupled with a 
system timing circuit.) Once a certain number of 
digital image signals have been accumulated in the 

75 image buffer 18, the stored signals are applied to a 
digital signal processor 22, which controls the 
throughput processing rate for the interpolation and 
recording section 4 of the camera. The processor 
22 applies an interpolation algorithm to the digital 

20 image signals, and sends the interpolated signals 
to a conventional, removable memory card 24 via a 
connector 26. 

Since the interpolation and related processing 
ordinarily occurs over several steps, the intermedi- 

25 ate products of the processing algorithm are stored 
in a processing buffer 28. (The processing buffer 
28 may also be configured as part of the memory 
space of the image buffer 18.) The number of 
image signals needed in the image buffer 18 be- 

30 fore digital processing can begin depends on the 
type of processing, that is, for a neighborhood 
interpolation to begin, a block of signals including 
at least a portion of the image signals comprising a 
video frame must be available. Consequently, in 

35 most circumstances, the interpolation may com- 
mence as soon as the requisite block of picture 
elements is present in the buffer 18. 

The input section 2 operates at a rate com- 
mensurate with normal operation of the camera 

40 while interpolation, which may consume more time, 
can be relatively divorced from the input rate. The 
exposure section 10 exposes the sensor 12 to 
image light for a time period dependent upon expo- 
sure requirements, for example, a time period be- 

45 tween 1/1000 second and several seconds. The 
image charge is then swept from the photosites in 
the sensor 12, converted to a digital format, and 
written into the image buffer 18 during a standard 
rate, which may, for example, correspond to a 

so standard video field or frame rate. The repetition 
rate of the driving signals provided by the control 
processor 20 to the sensor 12, the A/D converter 
16 and the buffer 18 are accordingly generated to 
achieve such a transfer. The processing throughput 
55 rate of the interpolation and recording section 4 is 
determined by the speed of the digital signal pro- 
cessor 22. 
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One desirable consequence of this architecture 
is that the processing algorithm employed in the 
interpolation and recording section may be se- 
lected for quality treatment of the image rather than 
for throughput speed. This, of course, can put a 
delay between consecutive pictures which may af- 
fect the user, depending on the time between pho- 
tographic events. This is a problem since it is well 
known and understood in the field of still video 
recording that a digital still camera should provide 
a continuous shooting capability for a successive 
sequence of images. For this reason, the image 
buffer 18 shown in Figure 1 provides for storage of 
a plurality of images, in effect allowing a series of 
images to "stack up" at video rates. The size of 
the buffer is established to hold enough consecu- 
tive images to cover most picture-taking situations. 

An operation display panel 30 is connected to 
the control processor 20 for displaying information 
useful in operation of the camera. Such information 
might include typical photographic data, such as 
shutter speed, aperture, exposure bias, color bal- 
ance (auto, tungsten, fluorescent, daylight), 
field/frame, low battery, low light, exposure modes 
(aperture preferred, shutter preferred), and so on. 
Moreover, other information unique to this type of 
camera is displayed. For instance, the memory 
card 24 would ordinarily include a directory signify- 
ing the beginning and ending of each stored im- 
age. This would show on the display 30 as either 
(or both) the number of images stored or the num- 
ber of image spaces remaining, or estimated to be 
remaining. 

The d ig ital signal processor 22 i nte rpolates 
each still video image stored in the image buffer 1 8 
according to the interpolation technique shown in 
Figure 2. The interpolation of missing data values 
at each pixel location follows the sequence shown 
in Figure 2; that is, first, the high frequency in- 
formation for the "missing green" pixels (i.e., the 
red and blue pixel locations) are interpolated to 
improve the luminance rendition and, secondly, the 
color difference information is interpolated at the 
high frequency locations by traditional bilinear 
methods to generate the other color of the CFA 
pattern. In the implementation shown in Figure 2, 
an adaptive interpolation technique is used in the 
luminance section 36 for optimizing the perfor- 
mance of the system for images with horizontal 
and vertical edges. "Missing green" pixels are 
adaptively interpolated either horizontally, vertically 
or two-dimensionally depending upon the gradient 
established between the green pixel locations in 
the vertical and horizontal directions around the 
"missing green" pixel. 

The first step is therefore to obtain gradient 
values, as represented by the block 40, in at least 
two image directions, for instance, horizontal and 



vertical directions. In each direction, the gradients 
comprise differences between luminance values at 
spatially-displaced pixel locations. The horizontal 
and vertical gradients of the luminance (green) 

s pixels are then evaluated relative to a threshold 
value in an evaluation block 42, and a preferred 
orientation for the interpolation of an additional lu- 
minance value is selected from the two directions. 
The luminance values are selected to agree with 

10 the preferred orientation, and the missing location 
is then averaged in block 44 from the two 
luminance(green) locations situated along which- 
ever gradient is below the threshold. (Four pixels 
are averaged when both gradients are less than or 

75 greater than the selected threshold level.) Then, the 
color difference is calculated in a chroma section 
38 by subtracting the interpolated green value at 
each chrominance pixel location (block 46) from 
the actual color pixel for that location. Finally, the 

20 color difference data for each luminance pixel loca- 
tion is interpolated using two-dimensional bilinear 
interpolation in the block 48. The data at this point 
may be reconstructed into its original components 
(RGB) or left as color-difference signals for further 

25 processing. 

In interpolating green data for pixel locations 
(red and blue) where green data is missing, the 
image gradients in horizontal and vertical directions 
are compared to a threshold level, thereby adap- 

30 tively adjusting the processing to suit the image 
data. Moreover, the threshold level is itself adjust- 
able to categorize the area of the image being 
interpolated. In this way, the interpolation may be 
adjusted to systems having various noise levels 

35 and to sensors having different modulation transfer 
functions. 

The four categories of image data determined 
from these gradient evaluations are: 

A. For (Horizontal Gradient > Threshold) and 
40 (Vertical Gradient > Threshold), the image areas 

represent high scene spatial detail. The four 
pixels are averaged since there is no evident 
scene structure. 

B. For (Horizontal Gradient < = Threshold) and 
45 (Vertical Gradient > Threshold), the image areas 

represent predominately horizontal scene struc- 
ture. Interpolation is performed by averaging 
horizontally to follow the scene contours and to 
minimize error caused by scene edges. This 
50 also maximizes image sharpness since scene 
contour information is not "averaged out" of the 
image. 

C. For (Horizontal Gradient > Threshold) and 
(Vertical Gradient <= Threshold), the image 

55 areas represent predominately vertical scene 
structure. Interpolation is performed by averag- 
ing vertically to follow the scene contours and to 
minimize error caused by scene edges. This 
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also maximizes image sharpness since scene 
contour information is not "averaged out" of the 
image. 

D. For (Horizontal Gradient < = Threshold) and 
(Vertical Gradient < = Threshold), the image 
areas represent little scene structure. The four 
pixels are averaged since this reduces the noise 
level of the interpolated pixel locations. This 
performs some "noise" reduction in the flat-field 
areas of the image. 
Although not limited to any particular CFA pat- 
tern, the interpolation technique has been applied 
to an RGB Bayer color filter array, as shown in 
Figure 3. One characteristic of the Bayer array is 
that it contains a repetition of RGB in a characteris- 
tic Bayer block form, as follows 

G R 
B G 

with two luminances (greens) always on one diag- 
onal, and two chromas (red and blue) on the other 
diagonal. The object of the interpolation is to obtain 
all three colors (RGB) for each pixel location. 

In Figures 4A-4D, the characteristic Bayer 
block is shown in four phases (as outlined) with 
respect to one pixel (circled) that is to be interpo- 
lated. Each pixel is identified by its row and column 
position. In each phase, the notation G(X,Y), R(X,Y) 
and B(X,Y) are values measured by the image 
sensor 12 for given horizontal displacements (X) 
and vertical displacements (Y) from the interpolated 
(circled) pixel. Adap G(X,Y) is an interpolated green 
value, using the following algorithm: 
For Gdiff-hor = jG(-1,0) - G(1,0)j 

Gdiff-ver = |G(0,-1) - G(0,1)| 

Threshold = Predetermined value 
If (Gdiff-hor < Threshold) and (Gdiff-ver < Thresh- 
old) 

Or (Gdiff-hor > Threshold) and (Gdiff-ver > Thresh- 
old) 

Then jAdap G(X,Y) = [G(-1.0) + G(1,0) + G- 
(0,-1) + G(0,1)J/4j 

If (Gdiff-hor < Threshold) and (Gdiff-ver > Thresh- 
old) 

Then jAdap G(X,Y) = [G<-1,0) + G(1.0)]/2j 
If (Gdiff-hor > Threshold) and (Gdiff-ver < Thresh- 
old) 

Then jAdap G(X,Y) = [G(0.-1) + G(0,1)]/2j 
As specifically applied to the four phases of Fig- 
ures 4A-4D, the processing algorithms are as fol- 
lows. 

Phase 00 (Fig. 4A) 
G = G(0.0) 

B = 0.5 * [B(0,-1) - Adap G(0,-1) + B(0,1) - 
Adap G(0,1)] + G(0,0) 



R = 0.5 " [R(-1,0) - Adap G(-1,0) + R(1,0) - 
Adap G(1,0)] + G(0,0) 
Phase 01 (Fig. 4B) 

R = R(0,0) 
s G = Adap G(0,0) 

B = 0.25 * [B(-1,-1) - Adap G(-1,-1) + B(-1,1) - 
Adap G(-1,1) + B(1,-1) - Adap G(1,-1) + B(1,1) - 
Adap G(1,1)] + Adap G(0,0) 
Phase 10 (Fig 4C) 
io B = B(0,0) 

G = Adap G(0,0) 

R = 0.25 ' [R(-1 ,-1 )- Adap G(-1,-1) + R(-1.1)- 
Adap G(-1,1) + R(1,-1) - Adap G(1,-1) + R(1,1) - 
Adap G(1,1)] + Adap G(0,0) 
75 Phase 11 (Fig. 4D) 

G = G(0,0) 

R = 0.5 * [R(0,-1) - Adap G(0,-1) + R(0,1) - 
Adap G(0,1)] + Adap G(0,0) 

B = 0.5 * [B(-1,0) - Adap G(-1.0) + B(1,0) - 

20 Adap G(1,0)] + Adap G(0,0) 

It should be clear from the several phases that 
green (luminance) is adaptively interpolated for the 
phases lacking green data (Phases 01 and 10) for 
the interpolated (circled) pixel. Red or blue 

25 (chrominance) is directly available from the sensor 
for phases 01 and 10. Red and blue (chrominance) 
is bilinearly interpolated in the other cases by refer- 
ence to color differences (R-G, B-G) from locations 
in the neighborhood of the interpolated (circled) 

30 pixels. The color differences are then summed with 
the luminance for the interpolated (circled) pixel to 
obtain red and blue. 

While the interpolation technique of the inven- 
tion has been described for use in an electronic 

35 camera, it may be incorporated as part of other 
apparatus for processing color data from an image 
sensor. For instance, the actual data from the im- 
age sensor 12 may be downloaded directly to the 
memory card 24, and the interpolation processing 

40 may take place subsequently when the card 24 is 
inserted into a player apparatus. The interpolation 
technique shown in Fig. 2 will then take place in 
the player. Such a "player" may in practice be a 
desk-top computer, and the interpolation technique 

45 of Fig. 2 is performed by a program in the com- 
puter. 

The invention has been described in detail with 
particular reference to a presently preferred em- 
bodiment, but it will be understood that variations 
50 and modifications can be effected within the spirit 
and scope of the invention. 

Claims 

55 1. Apparatus for processing a digitised image sig- 
nal obtained from an image sensor (12) having 
color photosites that generate a plurality of 
color values, but only one color value for each 
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photosite location, said apparatus comprising: 

means (18) for storing the digitized image 
signal as an image-wise pattern of luminance 
and chrominance values, wherein some 
photosite locations lack luminance values; 

a processor (22) operative with said storing 
means (18) for generating an additional lu- 
minance value missing from a photosite loca- 
tion by the interpolation of the additional lu- 
minance value for such locations from lumi- 
nance values at nearby photosite locations, 
said processor (22) including 

means (40) for generating gradient values 
from the differences between luminance values 
in at least two image directions; 

means (42) for comparing the gradient val- 
ues to one or more thresholds; 

means (42) responsive to the threshold 
comparison for selecting one of said at least 
two image directions as a preferred orientation 
for the interpolation of the additional luminance 
value; and 

means (44) for interpolating the additional 
luminance value from luminance values se- 
lected to agree with the preferred orientation. 
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interpolating each missing color value from 
like color values at nearby locations selected 
to agree with the preferred orientation. 

A method for interpolating missing luminance 
values from a digitised image signal obtained 
from a color image sensor (12) having 
photosites that generate a pattern of actual 
luminance and chrominance values, one actual 
value for each photosite location, said method 
comprising the steps of: 

obtaining gradient values from the differ- 
ences between actual luminance values in ver- 
tical and horizontal image directions; 

comparing the gradient values to a pre- 
determined threshold; 

selecting the image direction, in which the 
comparing step indicates scene structure, for 
the interpolation of the missing luminance val- 
ues; and 

interpolating each missing luminance value 
from actual luminance values at nearby loca- 
tions corresponding with the selected direction, 
and thereby with the scene structure. 



Apparatus as claimed in claim 1 wherein said 
one or more thresholds are adjustable. 



Apparatus as claimed in claim 1 wherein sated 
at least two image directions are horizontal and 
vertical directions. 
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Apparatus as claimed in claim 1 wherein said 
image-wise pattern of luminance and chromin- 
ance comprises chrominance values inter- 
spersed in a checkerboard pattern of lumi- 
nance values. 



5. Apparatus as claimed in claim 5 wherein said 
luminance color is green, and said chromin- 
ance colors are red and blue. 
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6. A method for interpolating missing color values 
from a digitized image signal obtained from a 
color image sensor (12)having photosites that 
generate a pattern of color values, one value 
for each photosite location, said method com- 
prising the steps of: 

obtaining gradient values from the differ- 
ences between color values in at least two 
image directions; 

evaluating the gradient values relative to a 
predetermined threshold; 

selecting, based on the step of evaluating 
the gradient values, a preferred orientation for 
the interpolation of the missing color values; 
and 



45 



50 



55 



6 



EP 0 632 663 A2 



§ 



O 
UJ 



CO 



< 



CM 




DIGITAL 
SIGNAL 
PROCESSOR 




PROCESSING 
BUFFER 




Is? 






. 0/5/ 






7 



EP 0 632 663 A2 



G 


R 


G 


R 


G 


R 


G 


B 


G 


B 


G 


B 


G 


B 


G 


R 


G 


R 


G 


R 


G 


B 


G 


B 


G 


B 


G 


B 



FIG. 3 



8 



V 



EP 0 632 663 A2 





-1 


0 


1 


(ROWS) 




/ 


G 


B 


G 




FIG. 4 A 


0 


R 


;© 


R ! 




(PHASE 00) 


-1 


G 


! B 


G \ 






(COLS) 














-1 


0 


1 


(ROWS) 




1 


B 


G 


B 




FIG. 4B 


0 


G 


®| 


G 




(PHASE 01) 


-1 


B 


G \ 


B 






(COLS) 














-1 


0 


1 


(ROWS) 




1 

0 


R 
G 


i G 

!® 


R \ 
G ! 




FIG. 4C 

(PHASE 10) 


-1 


R 


G 


R 






(COLS) 














-1 


0 


1 


(ROWS) 




1 

0 


i 6 

\ B 


R \ 

©i 


G 
B 




FIG. 4D 

(PHASE II) 


-1 


G 


R 


G 







(COLS) 



9 



